1. Field of the Invention
The invention relates to a three-dimensional image pickup art based on a triangulation technique for picking up a pattern projection image provided by applying pattern light to a measurement object from different directions by one or a plurality of image pickup device and obtaining a range data based on pattern change.
2. Description of the Related Art
As a technique of acquiring a three-dimensional shape, active vision and passive vision are available. The active vision includes (1) a technique of emitting laser light or a ultrasonic wave, measuring reflected light amount from an object or arrival time at the object, and extracting depth information, (2) a pattern projection method of using a special pattern light source such as stripe light to estimate an object shape from image information such as geometric change of an object surface pattern, (3) a method of forming contour lines based on moire fringes by optical processing and obtaining three-dimensional information. On the other hand, the passive vision includes monocular stereoscopic viewing for estimating three-dimensional information from one image using knowledge concerning object viewing, light source, lighting, shadow information and the like, and twin-lens stereoscopic viewing for estimating depth information of each pixel based on the triangulation principle.
Generally, the active vision has higher measurement accuracy than the passive vision, but often has a small measurement range because of limit of a light projection device and the like. On the other hand, the passive vision is a general-purpose technique and has less restriction on an object. The invention relates to the pattern projection method of a range finder of the active vision.
In the pattern projection method, reference pattern light is projected onto an object and an image is picked up from a different direction from a projection direction of the reference pattern light. The picked-up pattern image is deformed by the shape of the object. The observed deformation pattern and the projection pattern are related to each other, whereby range finding on the object can be conducted. The pattern projection method involves a problem of how to lessen erroneous correspondence in relating the deformation pattern and the projection pattern to each other and easily relate the deformation pattern and the projection pattern to each other. Then, various pattern projection techniques (space coding method, moire, color coding) have been proposed.
As a typical example of the space coding, a proposition disclosed in JP-A-5-3327375 will be discussed. An apparatus in the example has a laser light source, a lens system, a scanner, a camera, and a controller. The lens system shapes laser light into a slit form. The scanner applies and scans the shaped laser light to and over an object. The camera detects reflected light from the object. The controller controls the components.
Laser light application portions and laser light non-application portions form stripes on the object by the laser light scanned from the scanner. As the laser light is applied according to different patterns, the object is divided into N identifiable portions. It is determined which division portion each pixel of the image of the object picked up by the camera from different positions is contained in, whereby the shape of the object can be calculated.
In the example, to raise the resolution, it becomes necessary to perform laser scanning two or more times and pick up an image by the camera two or more times. For example, when a full screen is divided into 256 regions (256 resolution levels), it becomes necessary to pick up an image eight times. Thus, it becomes difficult to pick up an image of an object moving fast and further the image pickup system needs to be reliably fixed during scanning. Therefore, even if the apparatus itself becomes simple, it is difficult to easily pick up an image.
As a technique for decreasing number of pattern light projection times, a color coding method disclosed in JP-A-3-192474 is available. In the color coding, letting q and k be predetermined natural numbers of two or more, a coded pattern is controlled so that using q or more colors, two adjacent stripe light beams do not become the same color and color alignment produced by k adjacent stripe light beams appears only once. Stripe colors are detected from the observed image. Stripe numbers are acquired from the color alignment of the corresponding stripes. The stripe application directions can be calculated from the stripe numbers and the range data can be calculated as with the example of the space coding.
However, in the color coding, codes are restored from the coding pattern list and thus there is a problem of large calculation amount of restoring the codes. Further, in a case of dividing an area into 256 regions using three colors of R, G, and B, it is necessary to know alignment of eight stripe light beams surrounding the stripe whose code is to be known. Therefore, the color coding suits only for measurement of a shape in which stripes can be continuously long observed.
As a technique for easily restoring stripes and further projecting a coded pattern once, a space coding method disclosed in Japanese Patent No. 2565885 is available. In Japanese Patent No. 2565885, there are provided three or more kinds of gradation areas using light and shade of a ternary value or more, or three or more colors, or a combination of light and shade with color. Also provided is a multi-valued lattice plate pattern disposed so that at least three kinds of gradation areas contact one another at an intersection point of boundary lines of the gradation areas. Main codes, which conform to the kind and order of gradations, are generated at an intersection point of projected images resulting from projection of pattern to an object to be measured, to touch the intersection point. A combined code obtained by combining the main codes with each other is used as a featurecode for identification of the previous intersection point.
However, in the above described system, coding is broken depending on the object to be measured and it may be made impossible to correctly relate codes to each other. Projected pattern row may be recognized with missing pattern row photographed by a camera or an inverted pattern row may be obtained depending on the structure of the photographed object. It becomes difficult to relate because of change in the projected pattern and photographed pattern row depending on the shape of the object, the reflection factor of the object, etc.
In the color coding, this problem is circumvented by using a technique in which decoding is not performed for a pattern having a possibility of loss or inversion of stripe when stripes are grouped at the decoding time. In the space coding method, a two-dimensional pattern is used to decrease the possibility of the error described above, but the same error will occur depending on the object on principle. Therefore, the above-described system can provide excellent accuracy in photographing (image pickup) in special circumstances in a laboratory or in circumstances where the objects are limited, but the accuracy is degraded in general photographing (image pickup) circumstances where the objects are not limited.
The applicant proposed a range finder, which does not depend on the object by feeding back a projected pattern and generating new code (JP-A-2000-9442, JP-A-2000-65542, and U.S. Pat. No. 6,356,298 corresponding to JP-A-2000-65542). The proposition will be discussed below:
FIG. 20 shows the general configuration of a range finder disclosed in JP-A-2000-65542. FIG. 21 shows a positional relationship between a light source and image pickup devices of the range finder.
In the configuration shown in FIG. 21, a range finder includes three cameras 101 to 103 and a projector 104. Distances 11, 12, and 13 in the figure are made equal so that the distance relationship among the cameras matches. The first camera 101 and the projector 104 are placed so that optical axes match using a half mirror 105 as a beam splitter. The second camera 102 and the third camera 103 are placed on both sides of the first camera 101 and the projector 104 so that the second camera 102 and the third camera 103 differ from the first camera 101 and the projector 104 in optical axis. A distance between the optical axis at the center and that on either side is base line length L.
The projector 104 has a light source 106, a mask pattern 107, a strength pattern 108, and a prism 109. An invisible-region light source using infrared or ultraviolet light can be used as the light source 106. In this case, each camera is configured as shown in FIG. 22. That is, incident light 310 is split in two directions through a prism 301. One is incident on an image pickup apparatus (for example, a CCD camera) 303 through an invisible-region (infrared or ultraviolet) transmission filter 302. The other is incident on an image pickup apparatus 305 through an invisible-region (infrared and ultraviolet) cut-off filter 304.
A light source having a wavelength range capable of picking up an image regardless of a visible region or an invisible region may be used as the light source 106 shown in FIG. 16. In this case, a CCD camera of progressive scan type is used as the first camera 101 and the second camera 102 and the third camera 103 may be of any configuration. However, considering compatibility with the first camera 101, CCD cameras of the same configuration as the first camera 101 are desirable. A pattern is projected from the light source 106 and the three cameras (101 to 103) pick up an image at the same time. Each camera acquires an image in batch by obtaining the light passing through the filter 302, 304 (see FIG. 17) in the image pickup apparatus 303, 305.
The configuration of the range finder will be discussed with reference to FIG. 20. As shown in the figure, the second camera 102 stores brightness information provided by picking up an image in brightness value memory 121 and stores an image pickup pattern in pattern image memory 122. Likewise, the third camera 103 stores brightness information in brightness value memory 123 and stores an image pickup pattern in pattern image memory 124. The first camera 101 stores brightness information in brightness value memory 125 and stores an image pickup pattern in pattern image memory 126. To reference a coded pattern, which is prepared in advance, later, the projector 104 divides each slit into cells on a square lattice for storage in frame memory 127.
Using the stored image pickup patterns and brightness information, a three-dimensional image is obtained as follows: The following operation is common to both a combination of the second camera 102 and the first camera 101 and a combination of the third camera 103 and the first camera 101 and therefore will be discussed by taking the combination of the second camera 102 and the first camera 101 as an example.
In FIG. 20, an area division section 128 performs area division of an image pickup pattern picked up by the first camera 101. The area division section 128 extracts an area with the strength difference between adjacent slit patterns being a threshold value or less as an area 1 where light from the projector does not reach, and extracts an area with the strength difference between adjacent slit patterns being the threshold value or more as an area 2. A recoding section 129 recodes the extracted area 2 using the image pickup pattern stored in the pattern image memory 126 and the projection pattern stored in the frame memory 127.
FIG. 23 is a flowchart for recoding. First, each slit pattern is divided in a longitudinal direction for each slit width (step S11) to generate square cells. The strengths of the generated cells are averaged and the average value is adopted as the strength of each cell (step S12). A comparison is made between each of cells of the projection pattern and each of corresponding cells of the image pickup pattern with respect to the strength in order starting at the center of the image. Then whether or not the strength difference between the cells is more than a threshold value because of change in pattern caused by the reflection factor of the object, the range data to the object, and the like is determined (step S13). If the strength difference between cells is not more than the threshold value, the recoding is terminated for all picked-up image cells (step S17).
If the strength difference between the cells is more than the threshold value, whether or not the cell of the image pickup pattern is a cell having new strength is determined (step S14). If the cell of the image pickup pattern is a cell having new strength, new code is generated and assigned (step S15). If the cell of the image pickup pattern is not a cell having new strength, coding is performed using a slit pattern alignment of other part (step S16). The recoding is now complete (step S17).
FIGS. 24A to 24C show a coding example of a slit pattern; FIG. 24A shows a projection pattern coded based on a slit alignment, and 3 (strong), 2 (medium), and 1 (weak) are assigned as the strength. In FIG. 24B, the strength is changed in the third cell from the left and a new code appears. Thus, new code of 0 is assigned. In FIG. 24C, an existing code appears in the third cell from the left, the second cell from the top. Thus, recoding is performed in such a manner that the vertical alignment is [232] and the horizontal alignment is [131] as new code from the cell alignment. This recoding is equal to light projection of a complicated pattern such as a two-dimensional pattern to a part of the object shape rich in change and light projection of a simple pattern to a part less in change. This process is repeated for assigning unique codes to all cells, thereby performing recoding.
FIG. 25 shows an example of projecting a coded pattern onto a plate 606 placed in front of a wall 605 using cameras 601 to 603 and a projector 604. The coded pattern is a slit pattern shown in FIG. 26. At this time, areas 801 and 901 as shadows of the plate 606 occur in images provided by the cameras 601 and 602, as shown in FIGS. 27 and 28. In the example, on the surface of the plate 606, a slit pattern as shown in FIG. 29 is provided as a new coded pattern.
Referring again to FIG. 20, a decoding section 130 of the second camera 102 extracts the image on which the projection pattern is pick up from the pattern image memory 122 and divides the image pickup pattern into cells in a similar manner to that described above. Using the codes provided by the recoding section 129, a decoding section 130 detects the code of each cell and calculates slit angle θ from the light source based on the detected codes. FIG. 30 is a drawing to show a range calculation method in the space coding. Range Z is calculated according to the following expression (1) from the slit angle θ of the cell to which each pixel belongs, the x coordinate on the image picked up by the second camera 102, focal distance F of a camera parameter, and base line length L:Z=(F×L)/(x+F×tan θ)  (1)
The range Z is also calculated in a similar manner in a decoding section 131 of the third camera 103.
For the area 1 described above, the rage is calculated as follows: In the area 1, pattern detection based on the projected pattern cannot be executed and thus a correspondence point search section 132 detects a parallax using the brightness information read from the brightness value memories 121, 123, and 125 of the cameras 101 to 103, and calculates the range based on the detected parallax. For areas except the area 1, the range is calculated by performing the above-described operation and thus the minimum value of the range of the area 1 is obtained and the pixels, which can be related, are also limited. Using the limitation, the pixels are related to each other and parallax d is detected and using pixel size λ of a camera parameter, the range Z is calculated according to the following expression (2):Z=(L×F)/(λ×d)  (2)
With the range information provided by using the first camera 101 and the third camera 103 in combination according to the technique described above, range information of the area 801 as the shadow of the plate shown in FIG. 27 cannot be detected. On the other hand, with the range information provided by using the first camera 101 and the second camera 102 in combination, range information of the area 901 as the shadow of the plate shown in FIG. 28 cannot be detected. That is, the shadow areas are areas blocked by the plate that light does not reach, and cannot be measured according to the system. Therefore, range information is found for areas other than the shadow areas. That is, a range information integration section 133 in FIG. 20 acquires range information for all pixels of the image of the first camera 101 (FIG. 31), namely, pixels other than the shadows from the range information calculated in the pair of the first camera 101 and the second camera 102 and the range information calculated in the pair of the first camera 101 and the third camera 103. The range information provided by performing the described operation is related to brightness image of the first camera 101, for example, for storage in three-dimensional memory, whereby three-dimensional image pickup is performed.
The description of the contents of the related art (JP-A-2000-65542) is now complete.
The range finder described in JP-A-2000-9442, JP-A-2000-65542 projects a projection pattern coded according to a plurality of strengths and a plurality of wavelengths. At this time, the projection pattern is changed by the effects of brightness information, material, and the like of the subject and when a three-dimensional shape is calculated, an error occurs and an appropriate three-dimensional shape cannot be measured. Thus, the range finder includes the monitor image pickup device placed at the same principal point as the projection element, monitors change in a projection pattern based on measured object information, performs recoding, and measures a three-dimensional shape.
In the configuration, however, making the same the principal points of the projection element and the image pickup device is indispensable for recoding; to do this, abeam splitter such as a half mirror is used to match the principal points of the projection element and the image pickup device.
If a half mirror or the like is used to match the principal points, the light projected from the projection element is attenuated through the halfmirror. The pattern projected onto the measured object is attenuated in brightness through the half mirror and is input to the image pickup device. It is assumed that ideally the half mirror can distribute at 50:50 at an angle of 45 degrees. The measured object is observed with an attenuation of 25% as compared with a case where light from the projection element is projected directly on the measured object and the measured object is observed on the image pickup device. In the range finder in the related art, as shown in FIG. 21, the half mirror intervenes between the image pickup device intended for recoding and the image pickup device for shape measurement. Because of the effect of the half mirror, double brightness difference occurs on principle between the measured object observed on the image pickup device for shape measurement and the measured object observed on the image pickup device intended for recoding.
Further, in the pattern projected from the projection element, pattern unevenness is also caused due to optical passage difference before and after being split, film thickness unevenness of the application material of the half mirror and the like. A light scattering produced by dirt, dust, or the like on the half mirror also occurs in coaxial image.
Thus, if information with the projection pattern changed by subject information is recoded in the monitor image pickup device, the image is not provided strictly under the same condition as the image observed on the image pickup device for shape calculation and therefore precision degradation of shape measurement is caused.
Further, as for the dynamic range of image pickup device, if image pickup devices of the same characteristics are used, it is difficult to drive the monitor and measurement image pickup devices under the same condition, and degradation of precision of measurement caused by recoding occurs.